DOCUMENT RESUME 



ED 275 747 



TM 860 664 



AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 
PUB DATE 
CONTRACT 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



ABSTRACT 



St. John, Mark 

Evaluation Design: Selecting Methods. Guide Number 
14. Evaluation Guides Series. 

Northwest Regional Educational Lab., Portland, OR. 

Research on Evaluation Program. 

National Inst, of Education (ED) , Washington, DC. 

[85] 

400-80-0105 

17p. ; For other guides in this series, see ED 253 
952-954, ED 256 637-638, ED 257 629, and TM 860 
663-666. Printed on colored paper. 
Guides - Non-Classroom Use (055) 

MP01/PC01 Plus postage. 

Data Analysis; Data Collection; Evaluation Criteria; 
♦Evaluation Methods; Evaluation Needs; *Evaluation 
Problems; Evaluators; *Program Design; *Program 
Evaluation; Program Guides; 'Research Tools 



The choice of methods is part of the overall 
evaluation design process. The process consists of the following 
steps: (1) analyzing the problem context; (2) asking a few general 
questions; (3) selecting the methods (strategies) to use; and (4) 
selecting the specific techniques (tactics) to use. To operate 
successfully the evaluator needs to know many different kinds of 
methods, understand their purposes, appreciate their limitations, and 
use courage and imagination. While method reflects the strategies of 
the evaluator, techniques reflect the tactics. Techniques are 
specific and can be viewed as tools. They are used for gathering and 
analyzing data. The following conclusions are made about the planning 
and implementation of an evaluation: (1) the evaluation problem is 
undefined (there is no one unique solution); (2) successful 
evaluation design cannot be prescribed (at best, a few heuristics may 
prove useful); (3) the power and flexibility of the evaluator is 
increased as his or her repertoire of methods and skills increases. 
Various selecting methods, methods to help in the evaluation design, 
tools for gathering and analyzing information, and tools for 
communicating findings are presented in tables following the article. 
(JAZ) 
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THE NEED FOR A REPERTOIRE OF METHODS 



When three blindmen in India set out to discover the true 
nature of the elephant, they all found it to be quite 
different— one said it was like a wall; another found it like a 
snake; a third was sure that elephants were like trees. By 
looking in one way, in one place, each thought he had understood 
the whole. 



Eva lua tors have the same tendency. By identifying themselves 
with an approach or a method, they limit their senses and 
restrict their view. And since what they see depends on how they 
look, they are able (however accurately) to learn about only one 
aspect of the program. Thus, when evaluators adopt a one* 
dimensional approach, they end up in a position where they are 
able to claim "the program is like this. • If, on the other 
hand, evaluators have at their command a wide repertoire of 
approaches and methods, and if they know how to combine these 
methods appropriately and flexibly, then, at the very least, they 
will be able to say, "The program is like this, and this, and 
this . . 



But you may argue, "Of course it is better to have more 
information, and therefore more views of a program, but resources 
are limited. We have to choose; we need to decide which methods 
really work, and which are really feasible and useful in the 
situations we face." 



This is true, but it is not as absolutely true as we usually 
believe. That is, we do not always need to choose a single 
method for all situations. To understand this, consider, as an 
analogy, the way that a carpenter works. 



A carpenter does not choose whether a hammer or saw is more 
useful— he knows they are both tools of his trade, and they are 
equally and exclusively useful, depending upon the situation. 
Carpenters do not find a need to debate the relative merits of 
the hammer and saw. Nor do carpenters have a preference for one 
tool over the other; they don't try to use a hammer when a saw is 
required. 



Different evaluation methods (experiments, case studies, 
surveys . • •) can be seen to be like the carpenter's hammer and 
saw — they are the evaluator's tools. Evaluators would do well to 
think of themselves as artists and craftsmen, and take pride in 
learning to use a wide range of tools skillfully. As carpenters 
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do, evaluators can learn to eye a situation and know instinctively 
which tools will work. As their skill increases, evaluators can 
learn how to extend the range of use of their tools, and even how 
to combine their uses in innovative ways to accomplish more 
difficult tasks. 



Carpenters 1 tools have evolved over centuries of continuous 
use; consequently, modern carpentry tools are simple, elegant, 
and ideally suited for their uses. By comparison, evaluators 
have had a short history of tool development. 

Fortunately, other disciplines have developed and refined 
methods for evaluation in their own fields. For example, wine 
taster 8, film critics, accreditation agencies, investigative 
journalists, test drivers, and senate committees have all 
developed approaches that are potentially useful for the 
educational evaluator. For several years, the Research on 
Evaluation Program has been collecting and adapting these kinds 
of methods for use in the evaluation of educational, social, or 
health programs. K sizeable literature describing the evaluation 
methods and techniques of a wide range of disciplines now 
exists. For the evaluator who is willing to experiment and learn 
new skills, these methods can become very useful tools. 



GOIDBLZHBS FOR SELECTING METHODS 



The development of many new and varied methods is perhaps a 
mixed blessing for the evaluator. For, while it empowers the 
evaluator, it also makes the task of deciding what to do more 
difficult. When should the evaluator experiment with a new 
method? and which method? What is the basis on which choices are 
to be made? 



To answer these questions, it is useful to remember that the 
choice of methods is part of the overall evaluation design 
process. As outlined in Guide Number 12, the design process 
consists of the following stepst 

1. analysing the problem context} 

2. asking a few general questional 

3. selecting the methods (strategies) to use; 

4. selecting the specific techniques (tactics) to 
use. 



Analyzing the Context 



Like any other design process, an evaluation begins with an 
analysis of the problem context— that is, with an investigation 
of the program as it is, and with an attempt to learn what is 
important in the setting, in a typical case, an evaluator faced 
with complexity may become lost in all the factors of problem 
context, as shown in Figure 1. 



As the evaluator investigates each of these factors and 
begins to "get the lay of the land," a working plan (a design) 
begins to form in his or her mind. Moving from the general to 
the specific, the evaluator begins to form some general and 
guiding questions in his or her mind. These questions provide a 
focus and purpose to the evaluation. They prioritize what is 
important to study. Examples of such general questions include: 

1. How can we best understand what is happening in this 
program? 

2. How could this program be made to work better? 

3. What are the outcomes of this program? 

4. What important variations are there in the program's 
activities or effects? 

5. How worthwhile is the program? 
Example i 

An evaluator in a state evaluation agency is given the 
task of evaluating a fourth grade reading program. The 
predominant factors in the problem context are the 
requirements set by the legislature for evaluation and 
the political pressure of parental groups for basic 
skills improvement. Thus, the evaluator in this case 
nas little leeway— the general question that must be 
addressed is something like this: how much have the 
fourth grade students improved their reading skills 
over the year? 

Bxamplet 

A museum has been given funding to train middle school 
science teachers. The evaluator has been called in 
early in the program to assist in any way possible. 
Here the evaluation problem context is more loosely 
defined, and the predominant general questions that 
emerge aret (1) What is happening in the program? (2) 
Who are these teachers? and What are their actual 
needs? and (3) How can the practices of the program be 
improved? 



Figure 1 
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Selecting Methods 



When the evaluator begins to pose general questions, he or 
she is implicitly beginning to select responsive and appropriate 
methods. That is, certain methods are better than others for 
answering the relevant questions of the evaluation. In the 
previous example of the evaluator assessing the improvement of 
the reading skills of the fourth graders, experimental methods 
and achievement tests are obviously called for. The evaluator 
working with the museum would more likely use case study or 
journalistic methods. 



The choice of methods also suggests a stance or role that the 
evaluator may adopt. Usually, almost unconsciously, evaluators 
adopt the role of the experimenter and tester, in using other 
methods, evaluators may bring a different approach with different 
perspectives and values to the evaluation. The evaluator can 
adopt not only the techniques, but also the perspective of the 
discipline from which the method is derived. Thus, in using 
journalistic methods, the evaluator begins to think and act like 
a journalist; in using methods of operations research, the 
evaluator takes on the role of the efficiency expert. 



Techniques 

If methods reflect the strategies of evaluators, techniques 
reflect the tactics. Methods come from the parent discipline, 
and they are complete with rationale, perspective, and 
techniques. The techniques are more specific and can be viewed 
as tools— the evaluator ■s tools. Techniques are adaptive and 
flexible in their use. They are used for gathering and analyzing 
data, for organizing findings, and for presenting results. 
Interviews, t- tests, surveys, research briefs, and thematic 
analysis are all examples of techniques used in evaluation. 



Putting it all Together 

As -the evaluator gains an understanding of the evaluation 
context, general questions begin to emerge. These questions in 
turn suggest methods and approaches with which to structure the 
evaluation. Within the context of the methods, specific 
techniques are used by the evaluator. The chart below shows a 
few examples of how these elements go together to form 4 rough 
working plan. 



General Question (s ) 

What are the main 
issues for the 
program participants? 

How could the 
program be run more 
efficiently? 

Is this method more 



effective? 



/ 



What is it like to 
be in this program? 



Method(s ) Technique (s) 
Case Study Interviews 



Modeling 



Pre- post 
Control 

Story- 
telling 



Queue ing 
theory 



Analysis of 
variance 

Stream of 
consciousness 



Evaluator 1 s 
Stance 

Anthropolo- 
gist 



Operations 
Researcher 



Experimental 
Designer 

Storyteller 



This movement from general questions to methods and techniques 
is illustrated more fully in Table 1. Part A of Table 1 lists 
five general questions and associated methods that cover a wide 
spectrum of evaluation purposes. Part B of Table 1 suggests 
questions and methods that can help the evaluator in the design 
process itself. 

Table 2 lists .nore specific evaluation techniques (tools) , 
describing the discipline from which they arise and the purpose 
for which they are most suited. 



SUMMARY 



Because the evaluation process can be viewed as a problem in 
design, the following conclusions can be made about the planning 
and implementation of an evaluation: 

• The evaluation problem is undefined; there is no one 
unique solution. 

• Successful evaluation design cannot be prescribed; at 
best, a few heuristics may prove useful. 

• The power and flexibility of the evaluator is increased 
as his or her repertoire of methods and skills increases. 



Thus, the evaluator 1 s search for a best method or »n a best 
way to choose among methods is futile? Rather, the > <ator is 
left in a less well defined but freer world. To opet success- 
fully in this world, what the evaluator needs is a kt adge of 
many different kinds of methods, an understanding of ir 
different purposes, and an appreciation of their limi .ions— and 
then, in addition, a little courage and imagination . . . 
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Table 1 



A. Selecting Methods 



Is 



How Can We Best Un do r stand what la Happening In the program? 
What Is the nature or character of the program? 
What are the conditions and activities like? 
What are the central issues, themes, conflicts, trade-offs? 
What seems important? 



Useful Methods 

Investigative 
Journalism 



Case Study 



Storytelling 



Criticism 



Photography 



Exploratory Data 
Analysis 



Relevant Tools 

Interviewing 
Tracking 

Shuffling, circling 
Files and profiles 

Interviewing 
Field observations 
Qualitative analysis 
Aggregation techniques 



Oral histories 
Narrative techniques 
Stream of consciousness 



Thematic matrix 

analysis 
Connoisseur sh lp 
Composing techniques 



Photo interviewing 
Sampling techniques 



Stem and leaf displays 
Box plots 

Functional transfor- 
mation of data 



General Purpose 

Td discover that which 
is Important, but not 
immediately apparent 



References* 
8, 15 



To gain insight into a program 
by understanding the many 
facets of the actual (not 
ideal) functioning of the 
program 

To convey humanness and com- 
plexity of a program; to 
create images and establish 
mental connections 

To illuminate forms; to 
demystify; to enhance 
sensibilities 

To provide an artful repre- 
sentation of reality 

To capture, and portray images 
of reality; to illustrate 
themes or Issues; to deepen 
insights with visual Images 

To discover relationships not 
immediately apparent in 
Accumulated data; to select 
appropriate analytical 
methods 



22, 26 



5, 6 



25 



How Could This Program Be Made to Wtark Better? 
Are resources being used optimally? 
Where is there a critical lack of feedback? 
What are the barriers to Improvement? 
What are the critical weaknesses? 



Dseful Methods 

Operations 
Research 



Service Delivery 
Assessment 



Hearings 



(COAt. ) 



Relevant Tools 

Assignment model 
Transportation model 
Queuelng theory 

Qualitative methods 
Debrief ings 
Briefings 



Committee hearings 
Panel reviews 



General Purpose 



To maximise the use of 
existing resources 



To provide policymakers 
near-Immediate feedback 
ab$it the oondltlons and 
activities of the program 

To gather and share testi- 
mony from Involved and 
affected parties about 
the program 



References* 

3, 16, 17 

9, 10 
24 
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Table 1 continued 



Hit What are the Outcomes of the Program? 
What objectives are/are not net? 
What "side effects" does the program appear to have? 



Oseful Methods 

Achievement 
Tasting 



Relevant Tools 

Experimental designs 
Quasi -experimental 
designs 



General Purpose 

To assess if program has 
statistically significant 
effect on participants 1 
skills or knowledge 



References* 



Survey 
Research 



Questionnaires 

Surveys 

Interviews 



To assess the perceptions 
and feelings of individ- 
uals and groups about 
the program 



18 



Product 
Evaluation 



Critical competitor 
Systematic check for 
side effects 



To make comparative judg- 
ments about product 1 s 
overall quality 



19 



Exploratory 
Data Analysis 



Stem and leaf displays 
Functional transfor- 
mations 
Box plots 



To make a search for unan- 
ticipated results and 
patterns in accumulated 
data 



IVt What Important variations Are There in the Program's Activities ok Effects? 
To what extent are different groups affected in different ways? 
In what ways has the program varied over time? 

How do the program's resources, services, or outcomes vary geographically? 



Useful Methods 
Hearings 



Relevant Tools 

Coanittee hearings 
Jury methods 



General Purpose References* 

To gather testimony from 24 
advocates of different 
points of view 



Document 
Analysis 



Tracking 

Legislative history 



To trace over time the 
changes in the operations 
or character of a program 



2, 4, 23 



Geographic 
Methods 



Geocode analysis 
Trend surface analysis 
Social aree analysis 



To assess the distribution 
of program parameters over 
regional areas 



14, 21 



VII i How worthwhile is the Program? 

. Overall, how good is the program? 
Is the program cost-effective? 



Useful Methods 

Product 
Evaluation 



Cost Analysis 



Relevant Tools 

Needs assessment 

Cost analysis 
Synthesis procedures 



Feasibility studies 
Cost-effectiveness 

analysis 
Cost-benefit analysis 



General Purpose 



To come to an overall 
judgment about a pro- 
gram's quality; to aid 
in decision making 

To render questions of cost 
into useful forms; to 
generate information to 
aid decision making 



References* 
19 

12, 13, 20 



(Cont.) 
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Tabla 1 contlnuad 



B* Nathods to Halp in tha Daalgn of tha Evaluation 



It Sffit Should tha evaluation Ibcua On? 

What ara tha critical or pay-off lssuas? 

Mat dlmanslons ara Important to lncluda in tha study? 



Uaaful Hathoda Balavant tbola 

Product Evaluation Chaekliat 



Xmraotlgatlva 
Journaliam 



Caaa Study 



Doc want Raviaw 



Quick study 



Observation and 

in tar via* tachnlquaa 

Lagialativa history 
Contant analysis 



Ganaral Purooaa Rafarancaa* 

V» aid tha avaluator in 19 
Baking a oomprahanslva 
assaasaant of a program 



lb gat tha "lay of tha land" 
and ravlaw ralavant 
background 

99 gain insight into tha whola 
by studying a slngla part 

99 laarn tha historical or 
laglslatad lntant of a 
program } to dlaoovar 
program thaaws or 
charactar is tics 



8, 15 



lit Ho» Can Ha H&va from tha Qsnaral to tha gpaclflo? 
Do *a agraa on tha mianing of kay tanas? 
Bow spaclf ic do wa wish to ba? 



Usaful Hathoda 

Philosophical 
Analysis 



Balavant Ttaola 
Conoapt analysis 



Oanaral Purooaa Bafaranoaa* 

99 clarify thinking about 1, 7, 11 
ganaral and aba tract 
quastlons? to aaa how con- 
cepts function in languaga 
and thought 



*Saa rafaranca list at and of this gulda. 
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Table 2 
Tools for the Evaluator 



A. Tools for Gathering Information 



Tools 
Investigative 



In-depth 
interview 

Testimony 



Observation 



Document 
review and 
tracking 

Achievement 
tests 

Operational 
testa 

Surveys and 
questionnaires 

Photographs 



Methods 

Investigative 
Journalism 

Case Study 



Committee Hearings 
Panel Reviews 



Case Study 

Phenomenonology 

SOA 

Investigative 
Journalism 
Legislative History 

Experimental 
Design 

Product 
Evaluation 

Market Research 



Photography 



Purpose 

To confirm hunches; discover 
new leads 

To probe: to gain insight 



To gather evidence and 
viewpoints of different 
interests 

To obtain "snap shops" of 
reality; to discover 
patterns 

To substantiate inferences; 
to learn hictory of issue 
or program 

To determine if groups are 
statistically different 

To measure the qualities of 
performance 

To discover the distribution 
of opinion 

To capture images of reality 



(Cont.) 
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Table 2 continued 



i. Tools for Analysing Information 



****** 



Purposs 



Factor analysis 
t-tsst 



Cost analysis i 
fsasbility 
effectiveness 

utility 
benefit 

Opsrational 
analysisi 
assignment 
transportation 
queue ing 

■ok plot 
Function trans* 

formation 
Stem and lsaf 

display 



Statistical 
Analysis 



Cost Analysis 



Trsnd surface, and 
Social arsa 
analyses 

Thssuitlo 
aatrlx analysis 

Concept 
analysis 

Content analysis 
Tracking 



Debriefing 
Co n noisseur ship 

Hearings 

Juries 

(Cent.) 



Operations 
Research 



Exploratory Data 
Analysis 



Geographic 
Methods 



Literary 
Criticise 

Philosophy 



Document 
Analysis 



Service Delivery 



Crltlcli 



Government Commit- 
tee Hearings 

Legal Proceedings 
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To determine if observed 
differences are 
statistically significant 

To determine if programs are 
feasible, or to measure 
costs against results 



To determine maximum use of 
resources; to minimize costs 



To discover relationships, 
and patterns hidden in 
accumulated data 



To portray the spatial 
distribution of program 
variables 



To identify predominant 
themes 

To clarify thinking, 
language and ideas 

To substantiate themes; 
to substantiate a 
hypothesis 

To arrive at consensus 
of perceptions 

To offer personal , 
expert analysis and 
opinions 

To syntheslse evidence 
in an open public format 

To judge evidence in the 
form of adversary testimony 
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Table 2 continued 



C. Tools for Communicating the Findings 



Tools 

Narrative 
Prose 

Briefs 



Graphics 

Naps 

Pictures 

Oral 

Br ief ings 

Hear ings 
Vignettes 



Methods 
Storytelling 

Journalism 

Art/Design 

Geography 
Photography 



Service Delivery 
Assessment 

Committee 
Hearings 

Case Study 



Purpose 

To convey the reality, human- 
ness of program 

To convey highlights in 
headline form 

To translate information into 
clear, insightful, graphic 
form 

To illustrate relationships 
using mapping formats 

To use pictures to heighten 
sense of program reality 

To give oral presentation 
of findings 

To present all testimony 
and evidence publicly 

To present in writing typical 
illustrative scenarios 
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